%        File: paper.tex
%     Created: Sat May 31 08:00 PM 2008 P
% Last Change: Sat May 31 08:00 PM 2008 P
%
\documentclass[titlepage,12pt]{article}
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{listings}
\usepackage{gloss}
\usepackage{setspace}
\usepackage[urlcolor=blue,pdftex,linkcolor=black,citecolor=black,
   colorlinks=true,plainpages=false]{hyperref}
\usepackage[square]{natbib}


\pagestyle{headings}
\title{CSC465 : Tube Truck Protocol \\ \textsc{\large A Multicast
Anonymous Encrypted \gloss[short]{RPC} Protocol}}
\author{Daniel Drexler \and Robert A. Iannucci Jr.}
\date{\today}

\bibliographystyle{acm}

\makegloss

\begin{document}
   \pagenumbering{alph}
   \maketitle
   \clearpage

   \pagenumbering{roman}
   \begin{spacing}{0.9}
      \tableofcontents
   \end{spacing}
   \clearpage

   \pagenumbering{arabic}
   %\onehalfspace
   \section{Introduction}
   In 2007, the music sharing site OiNK was shut down for sharing
   copyrighted material. Using the log files from the OiNK servers,
   federal agents identified and arrested users, using the IP
   addresses assigned by their ISP as a means of
   indemnification~\cite{tf:OiNK}. The OiNK incident displays many
   of the inherent problems when using servers to communicate
   securely or anonymously. Servers and unicast routing both represent
   unique challenges in maintaining anonymity. Both simplify the
   process of communication, but leave information others can use
   to identify what was said and who said it.  To get around these
   limitations, we decided to create a protocol that uses multicast
   for message delivery and avoids servers entirely.  We have
   replaced servers with a democratic system where security is
   guaranteed if clients behave and transparency is guaranteed if
   they do not (i.e.  everyone will be able to detect noncompliance,
   and up to the point of detection, all communications are secure).

   This new protocol is called \gloss[Long]{TTP} and is named for
   Alaskan senator Ted Stevens, whose contributions to the Internet
   cannot be understated\footnotemark. Though new problems arise
   from Tube Truck's serverless multicast paradigm, it renders
   tracking the activities of individuals extremely difficult, which
   is beneficial to those who prefer to remain anonymous on the
   Internet. Tube Truck also incorporates extensive encryption and
   identity verification, ensuring that communication is neither
   observed nor spoofed. Finally, \gloss[short]{TTP} implements a
   form of \gloss[Long]{RPC} that allows groups of Tube Truck
   applications to exchange data automatically and invisibly --
   removing the need for the application writer to deal directly
   with the network layer.

   \footnotetext{The Senator's contributions can best be summed up
   in a quote, ``The Internet is not something you just dump something
   on. It's not a truck. It's a series of tubes. And if you don't
   understand those tubes can be filled and if they are filled,
   when you put your message in, it gets in line and its going to
   be delayed by anyone that puts into that tube enormous amounts
   of material, enormous amounts of material.'' -- Ted Stevens }

   \section{Protocol Use}
   We designed Tube Truck to be a few significant things to a few
   people.  It is not a general-purpose protocol, as there are many
   services which do not map to Tube Truck in the slightest. Tube
   Truck is meant to be: 

   \pagebreak
   \begin{itemize}
      \setlength{\itemsep}{-.2em} 
      \item Serverless 
      \item Anonymous
      \item Secure 
      \item Easy to use (from the programmer's perspective)
   \end{itemize}

   The last item makes the whole protocol somewhat tricky to
   implement.  We wanted any programmer to be able to write their
   application with minimal knowledge of Tube Truck's inner workings,
   especially with regard to multicast, \gloss{RPC}, etc.
   Ideally, using the Tube Truck API should involve nothing more
   than writing a few callback functions, registering an object
   (the client's state), and then initializing Tube Truck.

   The main interaction between Tube Truck and the application layer
   is via a `State' object.  The client's State is replicated through
   the network, such that every member of a Tube Truck group has a
   copy of that client's state.  The application then updates its own
   state, which causes \gloss[short]{TTP} to invoke \gloss{RPC}s
   to all other members of the group, which will cause their Tube
   Truck clients to update the State of the originator.

   All synchronization is taken care of by Tube Truck, without any
   special hooks into the client application.  Every member on the
   network is guaranteed an accurate reflection of all other clients'
   states (though they may be slightly out of date at any given
   time).  Tube Truck automatically retrieves \gloss[short]{RPC}s
   from the network and applies them to the appropriate replica
   states.  It then informs the application that it has received an
   update.

   The synchronization issues are not as big of a problem as they seem,
   considering the target programs (Games, Chat systems, `fire and
   forget' distributed computing).  Unfortunately, we are not able to
   guarantee bounds on the number of frames missed at any given time,
   though \gloss{TTP} is designed to guarantee that at any given time,
   States are valid, and that they will eventually resynchronize.
   
   With this in mind, let's explore some of the potential applications
   of \gloss[short]{TTP} across multiple domains of computing.

   \subsection{Multiplayer Games}
   Tube Truck's \gloss[short]{RPC} mechanism works very well for
   game clients, especially in a game in which each client is responsible
   for its own units. Each client would make calls describing their
   units actions (attacks, creation, etc.) and Tube Truck automatically propagates
   those changes to other clients. This allows the game
   to push considerations based on multiplayer protocols into the background.
   Many games also have problems with either the matchmaking service
   or losing the ability to moderate the game for one or more
   players. These problems generally become more severe as the servers
   grow in size~\cite{battlenet}.  \gloss[short]{TTP} avoids this
   by having neither a host nor a server. Any one player could have
   some sort of client-side failure and the other players could
   continue onward using an application-specific mechanism to handle
   time outs.

   The most compelling reason to use Tube Truck is the light financial
   burden. Using Tube Truck does not require any centralized hosting
   equipment and scales well to large player bases through the use
   of many Multicast Groups, hierarchically organized (\S\ref{sec:flex}).
   This avoids many of the problems that games have with growing
   user populations where servers are strained past the point of
   their design~\cite{battlenet}. As the user base increases, players would have to
   move through a larger and larger number of Multicast Groups to
   find one where games were offered -- but the underlying protocol
   would never require fundamental changes.  The organization of
   groups could be designed as an $M$-way tree, requiring, at most,
   an $O(log_M N)$ traversal to find a final game group in a game
   with $N$ clients net-wide.


   \subsection{BotNet / Legitimate Distributed Computing Task}
   Tube Truck's ability to obfuscate the identity of its users means
   that it would be a very good protocol for controlling networks
   of ``zombies'' (computers controlled illegally via unauthorized remote
   users). The emphasis on identification through cryptographically
   strong techniques would allow botnet controllers to control the
   computers they had compromised using public Multicast Groups.
   The bots would only respond to messages sent with the proper
   encryption.  Different groups of bots could be defined by having
   them join different Multicast Groups.

   This sort of distributed approach has already been taken by
   successful botnets, but to a lesser extent. Current examples use
   small groups of infected machines that only know about a limited
   number of other infected machines -- making the botnet hard to
   isolate and evaluate~\cite{zdnetStorm}. Using \gloss[short]{TTP},
   bots could coordinate without any knowledge of the location or
   identity of other infected computers. The use of public/private~key
   cryptography means that security companies trying to reverse
   engineer a worm that uses Tube Truck would only know what Multicast
   Group that worm used, not what other Multicast Groups are used
   or other infected clients. This level of encapsulation is achieved
   by avoiding central servers and unicast communication.  While the
   initial reaction might be to simply stop routing that Multicast Group, it's
   conceivable that a botnet could operate undetected on a Multicast
   Group also in use by legitimate services.

   On the more legal side of things, `consumer grade' distributed
   computing applications, such as SETI@Home and Folding@Home, use a
   model where they have many different computers compute the same task
   to statistically guarantee correct results.  They also have no hard
   deadline requirements for when work units must be
   completed~\cite{ScreenSave}.  These, and other `embarrassingly
   parallel' problems can be refactored easily in terms of Tube Truck.
   Different Tube Truck Groups represent data segments that need to be
   worked on, and hierarchies in those groups represent subproblems.  Strong
   signatures could be used to make sure that work done is coming from
   the person you think it's coming from.  One would have to set up a
   less anonymous system for authentication, however, and require that
   new work units only come from peers that are properly authenticated
   to have valid work units. 

   \subsection{Hierarchical Serverless Chat}
   Another potential application is a traditional form of chat, but
   tweaked in slightly a different way.  \gloss[Long]{IRC} is a
   commonly used chat system on the Internet.  In its basic form,
   a server hosts channels, and allows anyone to join a channel.
   Once joined to a channel, that user may communicate with other
   people in that channel.  \gloss[short]{IRC} servers are often
   set up with many channels, and in order to get to a specific
   channel, newcomers join the general chat channel, and then
   subsequently join a more specific channel on which to communicate.
   \gloss[short]{IRC}, in its basic form, applies no 
   encryption, and often the IP addresses of the joining clients
   are logged.  \gloss[short]{IRC} also requires a server to host
   all of the chat activity, and depending on the popularity of the
   host, the server's requirements can be quite substantial.

   The application programmer could employ Tube Truck to create a similar system, except
   without the need for servers, the ability to have nested channels,
   and for the identities of the individual users to remain completely
   anonymous.  By using Multicast Group addresses, people can join
   a chat session in progress, or create one of their own for other
   people to join in, without the setup time or resources of a
   server.  The routers between group members would be the only
   ones to know who is part of a particular chat session or not.

   Any member in the group could also announce other groups that they
   know about in their \gloss{IIS} packets (See \S\ref{sec:Impl}), allowing members of a group
   to form meshes of related channels.  Members of group $X$ that know
   about group $Y$ will likely find that other members in group $X$ will
   also be interested in group $Y$.

   In addition, all communications between members of a group would be
   encrypted, ensuring that any third parties would be unable to listen
   in without first joining the group (though Tube Truck's anonymity
   somewhat negates the usefulness of this feature).  One sticky point
   of this chat system would be the inability to `moderate' channels, or
   to kick/ban members whose presence is detrimental to the high quality of
   conversation most likely taking place in the channel (\`a la
   \gloss[short]{IRC}).  This ability in \gloss[short]{IRC} has proven
   to be invaluable to the integrity of channels.  On the other hand,
   Tube Truck would allow the offended parties to simply start a brand
   new channel with virtually no setup time or cost.

   \section{Challenges} 
   Because of Tube Truck's unique requirements, we were presented
   with several challenges, which we didn't initially anticipate.
   This section details those issues, and particularly why they are
   problematic for Tube Truck's design.  Some of these problems became
   obvious to us only after we had completed some of Tube Truck's
   implementation.  Our solutions to these problems are located in
   \S\ref{sec:Impl} and \S\ref{sec:Details}.

   \subsection{Remote Procedure Call} 
   \gloss[Long]{RPC}, is a widely-used method of \gloss[Long]{IPC}.
   Compared to other forms of \gloss[short]{IPC}, \gloss[short]{RPC}
   allows one machine to access resources, such as knowledge or
   processing time, on another machine, simply by making what appears
   to be a local function call.  The local function call actually
   sends a request over the network to the other machine, which
   processes the request, and then sends back an answer, which the
   local function then returns to the caller~\cite{wiki:RPC}. All
   of this happens seamlessly without the calling application having
   to know the internals of the network protocol. This original
   flavour of \gloss[short]{RPC} has been around since 1976 where
   it was proposed as an alternative to the typical Command/Response
   protocols that were being developed at the time, such as FTP,
   SMTP, and others~\cite{rfc0707}.

   Traditionally and currently, \gloss[short]{RPC} has been used
   in conjunction with unicast protocols such as TCP and
   UDP~\cite{rfc1831}.  The reasoning for this is quite simple.
   Since you want to get a specific result back, you need to have
   one specific, other machine that will answer you.  Unicast is
   perfect for this, because it very clearly establishes one computer
   as the requester, and one computer as the supplier.  This was
   fantastic for 1976, and is still quite useful today.  One feature
   (or flaw) of this method is that \gloss[short]{RPC}s cannot
   have side effects on the local machine in the same way that a
   regular local procedures can have side effects.

   Object Oriented programming, however, has side effects
   on almost every line of a program, and has become a very common
   programming paradigm in the years since 1976. In order to allow meaningful 
   \gloss{RPC}s in an object oriented environment, various solutions, such as Java's
   RMI, 
   % todo CITE 
   or CORBA  
   % todo CITE 
   have been formulated.  These solutions are both very heavyweight
   and feature-rich, but they still have flaws.  They're flexible
   and modular, allowing any manner of data to be passed back and
   forth to any object, but they still insist on there being a
   one-to-one relationship between caller and responder.

   With \gloss[short]{TTP}, we wanted to retain the application-facing
   simplicity of local function calls, but we also wanted to have those
   function calls be meaningful in a one-to-many environment.  We
   initially intended to let the application do all of its own
   communication formulation, simply letting \gloss[short]{TTP} be a
   carrier for arbitrary data, but this imposed many constraints on the
   client application which would have been difficult to debug and
   design properly, especially because we targeted languages such as C++ 
   and higher.  It would be extremely useful for users to be allowed to
   pass objects across the network, and so we developed an
   \gloss[short]{RPC}-like system.

   \subsection{Acknowledgements} 
   The traditional client-server model used on the Internet uses
   acknowledgement packets to inform clients when their data has
   been received. This works well because the client and server are
   communicating exclusively with each other. The acknowledgements
   make traffic much more reliable with low overhead. When sending
   data to multiple parties using unicast, this method continues
   to yield good results. Though the server will receive more data
   (from many clients acknowledging packets), computers that are
   set up to act as servers can be given more bandwidth and processing
   capacity~\cite{Ourbook}.

   However, when dealing with a multicast medium, generating one
   acknowledgement per packet would congest the multicast group.
   On unicast media, the number of packets on one connection roughly
   doubles when using acknowledgements one per data packet.  On
   multicast the number of packets rises with the number of receivers:
   for $N$ receivers, a traditional acknowledgement system would
   need $N$ acknowledgements, per packet. Using a system like this,
   on a multicast network with five clients, only one sixth of the
   packets would be data, and the rest would be acknowledgements.
   If $N$ clients transmit data, the network will have to carry
   $N^2$ acknowledgements. This is an extremely inefficient use of
   network resources, so we had to devise a different method for
   \gloss[short]{TTP}.

   \subsection{Group Membership}
   \label{sec:GroupMem}
   Presuming fully anonymous clients and then enforcing that any
   select group of clients have secure communication with each other
   begs the question of, ``How do clients, who have no prior
   identification, get to be members of the group?''  Because of the 
   anonymity requirement of Tube Truck, it is impossible to establish 
   if someone should or should not
   be in the group prior to their request to join\ldots to do so
   would require that that client be identifiable!

   Some previous systems have used a ``vouching'' method, where
   particularly trusted clients give `votes' or `credits' to those
   they know have good group participation characteristics.  The
   EDonkey Filesharing network is one such system, and has the
   servers record the ID hashes of clients, along with their
   participation metrics.  Clients who are being solicited for
   upload slots then ask the servers if they can trust the newcomers,
   and the server then ranks them~\cite{wiki:EDK}.  This is a fine
   method, but it doesn't work directly for Tube Truck, not the least reason
   is that we don't have any servers.  However, we took this idea and
   make it a bit broader, and simply left the voting process in the
   hands of the application.

   \subsection{Group Authority}
   A follow-on problem to Group Membership (\S\ref{sec:GroupMem}),
   is that once you let someone into the group, they must be able
   to speak for the group (if it were any other way, it would
   eliminate the serverless nature of Tube Truck).  This has the
   unfortunate side effect that if the group ever allows an
   untrustworthy client into the group, that client will have all
   of the keys necessary to sign for and communicate with the group,
   which means they can provide anyone else with the same information.

   As we discuss in \S\ref{sec:Anon}, true anonymity and no central
   authority come at a price. Tube Truck attempts to solve the problem
   by leaving `authentication' of individuals up to the application
   layer.  This does allow the application to layer a more secure form
   of identification/authentication (enforcing key pairs to be
   registered by a trusted third party, replacing the public/private key
   pairs with certificates signed by a known CA, etc.).  Unfortunately, for
   most applications, the default `authentication' scheme will be a
   simple yes/no vote from the users of the application, based on the
   self-declared user name of the new participant, along the lines
   of, ``Do you want \verb|AgentSmith117| to join?''  Certainly not the
   epitome of security.

   Even so, this doesn't address the problem that once someone is in,
   they are in for good. It is conceivable that we could implement multiple levels of
   authority, but the fundamental problem of, ``Who can I trust?'' still
   remains.  Multiple levels of authority would only delay the
   inevitable problem.  There might be some way to tie all of the group
   members' keys together, so that only when there is a true consensus
   (or a majority consensus), the group can assemble the appropriate
   key, but that is neither here nor there.

   \section{Systematic Implementation}
   \label{sec:Impl}

   \begin{figure}[htp]
      \begin{center}
         \includegraphics[width=\textwidth]{pictures/Center1}
      \end{center}
      \caption{Joining the Lobby of a Group}
      \label{fig:c1}
   \end{figure}

   Before we launch into the minutia of \gloss[short]{TTP}, we'll first
   look at how Tube Truck works as a whole.  The
   very first step, once you start a Tube Truck-enabled application, is
   to join a Multicast Group (probably the \gloss[Long]{WeKMuG} for that
   application) as a ``Lobby'' participant.  A lobby participant, can announce 
   his/her presence, establish an identity (public/private~key pair), listen for 
   other members in the lobby, or request to join a Tube Truck Group (a group
   which communicates via encrypted \gloss{CHEEZBURGER}s.)  Figure
   \ref{fig:c1} illustrates this process.

   As you can see, the newcomer first sends an \gloss{OHAI} to the
   Multicast Group, the
   \gloss[short]{WeKMuG} in this case.  There are two possible outcomes from this.
   The first is that the newcomer's self-generated ID hash (a
   function of his/her public key) collides with that of someone
   else who is already in the lobby.  In this case, the
   newcomer is rejected with a \gloss{NOWAI} packet.  The newcomer
   would probably re-generate his/her key at this point and try
   again.

   If the ID hash is unique in the Multicast Group, the members respond
   by sending self-identification packets, \gloss{IIS} packets, which
   contain any application-specific identity information (user name,
   game status, etc.), as well as their Public Key, and any Tube
   Truck groups that they know about, along with those groups' Public
   Keys. Once this self-identification process occurs, the newcomer also
   self identifies with an \gloss{IIS}, as well.

   \begin{figure}[htp]
      \begin{center}
         \includegraphics[width=\textwidth]{pictures/Center2}
      \end{center}
      \caption{Joining a Tube Truck Group}
      \label{fig:c2}
   \end{figure}

   After a newcomer has joined a Multicast Group in the Lobby state,
   they are free to stay in the Lobby state as long as they like.  Most
   clients will, however, attempt to join a Tube Truck Group, and will
   do so by sending a \gloss{CANHAS} packet to the Tube Truck Group (as
   identified by its public key).  The Tube Truck Group members will then vote
   amongst themselves using encrypted \gloss{HASFLAVR} packets.  Once
   the group has reached a consensus, it falls on the group member with
   the lowest ID hash to disseminate the final result to the newcomer.
   If the person with the lowest ID fails to issue the answer after a
   timeout, the responsibility falls on the person with the next highest
   ID, and so on.

   If the group votes to disallow the newcomer, the newcomer is
   sent a \gloss{LAWL} packet signed by the group.  If the group
   votes to allow the user into the group, a \gloss{DOWANT} packet
   is issued, which contains the group's private key and symmetric
   encryption key, which is signed by the group and encrypted with
   the newcomer's public key, ensuring safe and authentic delivery. Once
   a newcomer has successfully received the keys for the group, they
   send out a \gloss{CHEEZBURGER}, with all of the acknowledgement
   numbers for the group members set to zero\footnotemark.  This triggers group
   members to post their full state to the Multicast Group, so that the
   newcomer can get up to speed.  Once the newcomer has fully
   synchronized all the mirrored states of the other clients, he/she
   begins to post \gloss{CHEEZBURGER}s to the group as well.  The
   newcomer is now a full participant in the Tube Truck Group.

   \footnotetext{Every CHEEZBURGER that is sent out contains a list of
   $\langle$ID hash, Ack Number$\rangle$ tuples, which represents implicit
   acknowledgements of all the other members in the group.  Ack numbers
   that are out of range of the histories that each client keeps (in the
   case of missed packets), automatically generate a full-state update.}

   \section{Implementation Details}
   \label{sec:Details}
   Tube Truck's underlying design has several interesting details.
   While the whole system is quite complex, we have extracted
   particularly applicable portions to discuss in this section.  

   \subsection{Remote Procedure Call}
   In order to allow clients to seamlessly make a \gloss[long]{RPC}
   over the network with the minimal amount of manual manipulation
   of their data, Tube Truck implements a C++-compatible
   \gloss[short]{RPC} system which can encapsulate whole C++ objects
   with a minimal amount of programmer effort.  In the
   application-overridden State object, the application programmer
   elects which methods, either \gloss[Long]{POD}, or full C++
   member functions, to be \gloss[short]{RPC} targets
   by registering them in the object's constructor.  Once this
   registration has taken place, the application programmer may
   call these methods by invoking the \verb|remoteCall()| method, passing in
   the method to invoke, as well as what arguments with which to invoke
   it.

   \lstset{language=C++,basicstyle=\footnotesize,tabsize=3,captionpos=b,
   caption={A user Serializable object.}, label=code:Serializable,
   frame=tb, belowskip=2em, aboveskip=2em}
   \lstinputlisting{serializable.cpp}

   The \verb|remoteCall()| method fully supports C++-style implicit
   construction, and will build the requisite parameter objects
   before serializing them and passing them over the network.  The
   only requirement on the application is that any parameter objects
   be \verb|Serializable|, \gloss[short]{POD}, one of the STL
   containers with \gloss[short]{POD}, or one of the STL containers
   with \verb|Serializable| data.  The \verb|Serializable| `interface'
   simply requires a \verb|toVec()| and a \verb|fromVec()| method,
   which, in turn, may call similar methods on member data, or use
   Tube Truck's \verb|serialize()| and \verb|unserialize()| method
   to serialize \gloss[short]{POD} member data.  An example user
   object is in Listing~\ref{code:Serializable}.

   \lstset{caption={A sample derivation of the TTP::State object},
   label=code:State} 
   \lstinputlisting{state.cpp}

   The user-derived State example in Listing~\ref{code:State} shows
   the level of interaction that a user program needs in order to use
   Tube Truck's RPC interface.  The user application controls
   everything up to the \verb|remoteCall()|, whereupon Tube Truck takes
   over and ensures that all other clients on the network will have
   that method called on their machines correctly, in local order for
   this client, and without duplication (in the case of retransmission
   for any reason).

   \subsection{Anonymity}
   Anonymity is one of the main goals of Tube Truck, and we use
   multicast as a means to accomplish this. Because multicast does not
   need an end-user specific address to deliver messages, it makes it
   much easier to obscure the sender's identity than with unicast. Instead,
   messages sent to a Multicast Group are delivered to all members
   of that group regardless of origin, without any clear method of tracing a message back
   to any particular end host. 
   
   Some multicast protocols {\em do} rely on the
   unicast source address of the multicast client in order to prune away
   branches of the multicast tree when sending a message, to ensure
   that the message doesn't slosh back and forth across the network.
   However, using a spoofed but unique source IP address circumvents this
   nicely~\cite{rfc1075}.  Ideally, Tube Truck packets should also use a
   spoofed MAC address to avoid identification by hosts on the same
   physical \gloss[short]{LAN}.

   After taking these measures, the only record of group membership
   associated with a particular computer is the multicast group
   membership information stored in the router closest to that
   computer. The process of connecting a human user of \gloss[short]{TTP}
   to messages exchanged in a Tube Truck Group is involved and
   difficult. First, one would have to circumvent the cryptography
   used, either through social engineering or cracking.  Second, the
   interested third party would have to track the user's packet
   flow back to the edge router the user registered with when joining
   the multicast group (difficult, as that information isn't included
   in the multicast packets and routers have no reason to keep or
   log packets)~\cite{Ourbook}. Finally, the third party would have
   to associate the spoofed IP address supplied by the client with
   his/her actual IP address. The third party could then take the
   normal steps to discover the user's `real world' identity
   (contacting the ISP of the client, etc.).

   Tube Truck version 1 (unfortunately) simplifies this process as
   the source address is not spoofed. An interested third party
   would only have to complete the first step to attach a real user
   to information transmitted over \gloss[short]{TTP}.  This was
   not originally planned, but adding spoofed MAC and IP addresses
   would have greatly increased the complexity of an already complex
   project, and for the sake of development time, this requirement
   was dropped.

   \subsection{Security}
   \gloss[short]{TTP} uses a multi-tiered approach to guarantee
   that communication is both verified and unobservable. Each client
   has a public/private key pair used for identification and group
   admittance. Also, each Tube Truck Group has its own pair of
   public/private keys used to sign communication with prospective
   clients and a symmetric key used for inter-group communication.
   
   \subsubsection{Identity}
   The ``real world'' identity of users on \gloss[short]{TTP} is
   not important.  All systems of identification are concerned with
   making sure that a user or group is the same user or group they
   either saw advertised or were communicating with previously.
   The only difference in Tube Truck is that there is no special
   importance attached to any one identification number--a user
   cannot reserve identities. This approach leads us to use our
   self-generated cryptographic keys as our identifying information.
   We could have instituted another form of identity, but until
   that other form was somehow cryptographically verified, it would
   be unreliable.  Instead, we choose to push all questions of
   human-friendly identification to the application level. We
   use a two-tier system for communication, which is reflected
   by our two forms of packets: unsigned (for Lobby communication), and
   signed (for in-group communication).  Unsigned
   packets are used where identity is unverifiable, or where a
   falsified packed would not have undue negative effects.

   \gloss{OHAI} and \gloss{NOWAI} packets are both unsigned.
   \gloss{NOWAI} packets only affect clients that have no established
   identity in a Multicast Group. Because \gloss{NOWAI} packets
   come from hosts with the same ID hash (not identical keys),
   signing would be ineffective for verification (as no communication
   has occurred besides the \gloss{OHAI}) (See Appendix \ref{sec:pckts} for more
   details on packet types).

   Signing the data in an \gloss{IIS} packet does not make sense
   because the transmitting client could send false information.
   Even if the packet arrived unaltered, the user or application
   still has to trust the information it received in the \gloss{IIS}.
   Because keys are generated on connection, all clients are on the
   same level and all information from \gloss{IIS} packets is treated
   equally. It doesn't make sense to guarantee the contents of a
   message isn't altered when anyone can send a message and have
   it be considered with the same level of trust.

   A third party could spoof the \gloss{CANHAS} packet by
   placing a users public key and ID hash in the packet. However,
   there's nothing invalid about the events that follow--that user
   would either be admitted or rejected from the group. Just because
   the user being voted on did not initiate the vote doesn't
   invalidate the process. The user whose information is transmitted
   is free to join the group if they're accepted.

   All other packets are signed, either by a Tube Truck Group (in
   the case of \gloss{DOWANT}s and \gloss{LAWL}s) or an individual.
   These packets are signed to avoid trusted data (application data,
   Tube Truck Group secrets) from being modified or exposed. The
   only packet that is signed, but not encrypted, is the \gloss{LAWL}
   packet. Signing the \gloss{LAWL} packet ensures that only the
   members of a Tube Truck group can reject applicants to that
   group.

   \subsubsection{Encryption}
   There are three packet types that are encrypted: \gloss{HASFLAVR}, \gloss{DOWANT}
   and \gloss{CHEEZBURGER}. All of those packets are also signed. All of
   these packets are encrypted to hide Tube Truck Group information
   from the greater Multicast Group. The data that each packet has
   encrypted is covered in the General section, this section will
   be concerned with the strength and handling of the encryption.

   \gloss[short]{TTP} version 1 uses BlowFish and RSA encryption.
   These protocols were selected because they are well known standards
   for symmetric key and public/private key encryptions. Any similar
   protocol could replace either. Our RSA encryption uses 1024 bit
   keys, the shortest keys that are considered safe~\cite{wiki:RSA}.
   BlowFish uses 160 bit keys and 80 bit initialization vectors.
   Packets encrypted with BlowFish include the initialization vector
   directly before the cyphertext.

   The only point where keys are exposed is when a \gloss{DOWANT} packet
   is sent to admit a user to a group. The \gloss{DOWANT} packet contains
   the group secrets, but is encrypted with the users public key.
   Because a users private key is never sent over the wire, the
   vulnerabilities come from cracking the encryption or from
   side-channel attacks. \gloss[short]{TTP} does not take any extra
   measures to counter any out-of-band attacks beyond whatever
   protections are built into OpenSSL.



   \subsection{Flexible Topology}
   \label{sec:flex}
   Tube Truck is designed around the idea of a ``\gloss[Long]{WeKMuG}''.
   On startup, each application joins a
   specific Multicast Group.  However, this group is probably not
   the best place to form Tube Truck Groups -- if all users of
   \gloss[short]{TTP} communicated on one Multicast Group, clients
   would receive a lot of traffic they're uninterested in. To deal
   with this, each client has the ability to advertise other Multicast
   Groups it knows about along with public keys for Tube Truck
   Groups in that Multicast Group. Clients in any of the Multicast
   Groups in these advertisements also have the ability to advertise
   additional Multicast Groups and Tube Truck Groups.

   This tiered structure allows users to have some control over how
   much traffic they have to deal with and size groups appropriately
   for the application in question. A game running Tube Truck might
   want a small Multicast Group to choose an uninhibited Multicast
   Group to minimize traffic unrelated to their game.  A chat
   program, on the other hand, could use a central Multicast Group
   as a general chat and have other users join smaller Multicast
   Groups to talk about more specific topics. The large number of
   IPv6 multicast addresses makes space limitations unlikely in the
   foreseeable future.


   \section{Flaws}
   Tube Truck is a young protocol.  So young, in fact, that it still has
   several significant flaws in its underlying design.  The protocol's
   authors note the known flaws in Tube Truck here.  Some of these flaws
   are simply due to Tube Truck's basic nature, but some of them are
   deeply crippling to what Tube Truck attempts to accomplish 
   (\S\ref{KeyExch}). 

   \subsection{Anonymity}
   \label{sec:Anon}
   \gloss[short]{TTP} was designed to ensure anonymity whenever
   possible.  However, there are some very good reasons to have a
   persistent form of identification. The problems that arise from
   being fully anonymous are different from those that arise from
   having identity, but can be generally summed up as: Harder to
   control resource usage, easier to protect data.
   
   \subsubsection{Wasting Group Resources}
   There are two points in the sequence of Tube Truck packets used
   to establish an identity in a Multicast Group and Tube Truck
   Group that are open to flagrant abuse. Both of these points can
   be exploited because it is easy to generate a new identity on
   Tube Truck. The unicast Internet requires users attach to
   themselves to IP addresses controlled by a hierarchy. A single
   client can not  use any IP address -- they are often limited to
   one or the range of their Internet Service Provider.  \gloss[short]{TTP}
   users have no such limitation.

   Any 32 bit number is a valid ID hash, and so clients can over
   four billion identities and send \gloss{OHAI} packets containing those
   ID hash values. Aside from the massive bandwidth usage for all
   clients, a significant amount of processing power would be spent
   sending replies to all such requests. Also, the memory required
   to cache all the apparent clients generated by so many \gloss{OHAI}
   packets is not insignificant.  There is no way to differentiate
   from this malicious traffic and legitimate traffic.  If the
   multicast protocol uses includes source IP address, \gloss{OHAI} packets
   with many different ID hashes from the same source IP could be
   ignored. However, this just means that the attacker needs to
   spoof a new IP address before continuing.

   The same problem affects \gloss{CANHAS} packets. Each \gloss{CANHAS}
   starts a vote, which can take a non-trivial amount of time.
   Memory cache is more a more significant problem here than it was
   with \gloss{OHAI} packets. Each vote made by the group is
   remembered so the clients do not have to vote again in the event
   of packet loss.  The same semi-solution to this problem exists,
   but because \gloss[short]{TTP} is essentially anonymous, this
   is an attack that has few countermeasures in the current version
   of Tube Truck.


   \subsubsection{Denying Access to Clients}
   \label{sec:Rej}
   Another vulnerability in the protocol comes when a new client
   tries to establish its identity as unique with an \gloss{OHAI}.
   Because ID hashes are 32 bits instead of 1024 bits, there are a
   great number of collisions. Therefore, a client is not required
   to have the same public key to send a \gloss{NOWAI}, just the
   same ID hash.  A malicious client is free to \gloss{NOWAI} every
   single request that is sent to a Multicast Group. This attack
   will probably fail eventually because an \gloss{IIS} packet will
   eventually beat out a \gloss{NOWAI} packet, changing the state
   of the applicant and causing it to ignore the \gloss{NOWAI}, but
   it still denies the client prompt service.  This could be remedied
   somewhat by forcing the client sending the \gloss{NOWAI} to
   include its public key with the \gloss{NOWAI}, and perhaps signing
   part of the packet with its private key. This would not stop the
   practice, but it would take the malicious client time to generate
   a new public/private~key pair each time a new client sent an
   \gloss{OHAI}. Hopefully the delay would allow an \gloss{IIS} to
   reach the new client.

   A second method for denying access comes at the end of the process
   of admitting a user to a Tube Truck Group. Because all communication
   is from the group to the new user could potentially be sent by
   any member of the group, any member of the group can reject the
   applicant. However, this problem is more closely related to
   issues surrounding the lack of a central authority.


   \subsubsection{Identity usefulness}
   A final, overall problem comes when a new client sends a
   \gloss{CANHAS} to a Tube Truck Group. The only thing that group
   knows is the public key contained in the \gloss{CANHAS} (and any
   information from \gloss{IIS} or \gloss{OHAI} packets). These are
   just randomly generated numbers that are totally arbitrary --
   they have no meaning to the users already in the Tube Truck
   Group. The client applying to a group could be a new client, or
   a client that was rejected thirty seconds ago. This flaw leads
   to the attack of sending many spurious \gloss{CANHAS} packets
   to a group.

   In the current version of \gloss[short]{TTP}, there is one
   solution to this problem, but it's a poor one. Using some out
   of band exchange method, users that wish to speak anonymously
   exchange public keys. Then, the users enter a list of ``safe''
   public keys. These public keys are the only keys that will be
   granted admittance to the users Tube Truck Group. Of course, a
   malicious member of the group could block entry to the WekMug
   by pretending to take all those public keys - but that is a
   subset of the problems introduced by having no admin and unrelated
   to the usefulness of identity.

   A more elegant solution would be to include application level
   data with \gloss{CANHAS} packets -- such as a password or other
   identifier.  This gives applications some identity beyond their
   public key and can form the basis for an automatic denial policy.
   A ``safe list'' of public keys will always be the most secure
   method, but passwords and other weaker methods of identification
   can be useful as well.


   \subsection{Lack of Central Authority}
   At the end of the process for admitting a new member to a Tube
   Truck Group, the applicant is sent either a \gloss{LAWL} or a
   \gloss{DOWANT} based on the votes from the group. However, the
   lack of any authority means that every user must be able to
   accomplish this. If the user charged with this responsibility
   were to suddenly disconnect, the group would no longer be able
   to add new members if they were not equally able to add people
   to the Tube Truck Group. The problem that arises is that nothing
   prevents clients from revealing all of the groups secrets to
   anyone who asks. Even if a use is rejected, one of the members
   of the Tube Truck Group could send the new user a \gloss{DOWANT}.
   Conversely, a user that was approved by the group could be sent
   a \gloss{LAWL}.

   This is part of a larger problem -- once a user is granted access
   into a group, they have what would be considered ``admin'' level
   powers in any other protocol. All users can perform all
   administrative tasks in a group. Within Tube Truck Protocol,
   this is somewhat mitigated by the shared medium of multicast:
   If a user gives away the Tube Truck Group's secrets, the other
   group members can probably tell what happened.  However, if
   out-of-band methods of communication or specially crafted packets
   are used, detection is difficult: It would be trivial to send
   an \gloss{IIS} packet that appears corrupt, but actually includes
   group secrets. These problems are unavoidable if central authority
   is to be avoided, as Tube Truck tries to do. It is, however, not
   a problem unique to Tube Truck -- by necessity, all encryption
   schemes that allow two entities to act on the same level must, at
   some point, give their secrets to that other entity.  Since that
   other entity is not us, we can not inherently trust them and there is
   the opportunity that we just gave the secrets to someone we didn't
   want to.

   This lack of central authority also means that many traditional
   admin controls in group situations are not available. Though
   clients can kick clients out of a Tube Truck Group, the process
   is not satisfactory. If clients decide to kick a client, they
   form a new Tube Truck Group with a ``safe list'' of public keys.
   This ``safe list'' contains all the keys in the old group aside
   from the ejected user. However, if this new group accepts new
   clients, there is no reason the ejected user wouldn't re-apply
   with a different key.  Thus, the new group must refuse all new
   clients. The functional requirements behind actions like kicking
   a user make them unsatisfactory in Tube Truck Protocol. What's
   the point of kicking an annoying user from a group chat if no
   one else can join that chat as a result?

   The final problem with equally shared power comes from the
   vulnerability in democracy. Users are free to run as many clients
   as they want -- their Tube Truck identities aren't tied to their
   physical identity or IP address. They could then have as many
   clients as required join a single Tube Truck Group (the likelihood
   of acceptance growing with each of their slave users being
   accepted).  Once they have a majority in the group, that user
   effectively controls the Tube Truck Group. They can carry any
   vote made within the group without the assistance of other users.
   Without some sort of ``safe list,'' this sort of attack is
   difficult to prevent, as there are no features that distinguish
   a ``slave'' client from a legitimate client.
   
   \subsection{Key Exchange}
   \label{KeyExch}
   The process joining a group is somewhat vulnerable to man-in-the-middle
   attacks. Similar to the method for attacking plain Diffie-Hellman
   using man-in-the-middle, Eve would have to be able to intercept
   messages from Alice while Alice is trying to join Bob's
   group~\cite{DiffHellman}.  When Alice sends her \gloss{CANHAS}
   packet, Eve replaces Alice's public key with one known to Eve
   (this could create problems if the ID hash from Eve's selected
   public key collides with another ID hash in Bob's group).
   Eve-Alice's request for entry is voted on by Bob's group, and
   assuming they accept Eve-Alice, they send a \gloss{DOWANT}
   encrypted with Eve's public key. Eve then decrypts the message
   using her private key, gets the shared secrets, and encrypts it
   with Alice's public key. Alice then receives the packet.

   Unlike Diffie-Hellman however, Alice can detect that the packet
   has been tampered with. Because packets are signed after they
   are encrypted, the key which encrypts a packet matters. The
   packet, signed by Bob's group's private key when encrypted with
   Eve's public key, would not verify after being re-encrypted with
   Alice's public key.  Additionally, this attack is not as harmful
   as it would be on other protocols. Eve's \gloss{CANHAS} is viewed
   as any other \gloss{CANHAS} packet -- any identity that might
   be attached to Alice's public key is lost. There is virtually
   no difference between this method and Eve sending a \gloss{CANHAS}
   without intercepting Alice's packet. In effect, this attack
   merely prevents Alice from entering the group, but cannot steal
   keys given to Alice.  In fact, if Alice were to use the group
   secrets sent by Eve, she would be ``stealing'' those secrets:
   Alice's key was never voted on by the group so she was never
   approved.  This technique is similar to the Station to Station
   protocol developed in '92 by Diffie et.~al.~to address the
   problems presented by the original Diffie-Hellman key exchange
   model~\cite{diffie92authentication}.

   \subsection{Protocol State Complexity}
   One major disadvantage of \gloss[short]{TTP} is the vast amount of
   state which must be retained on each client.  In addition to the
   overhead of having replicated State objects for each other member in
   the group, the protocol must retain enough information to ensure that
   the voting process works as anticipated (keeping track of what
   results have been heard, and if the group member who was supposed to
   send out the vote result did, etc.).  In addition, in order for the
   acknowledgement system to work, each client has to retain the
   acknowledgement numbers from all clients to all clients.  This is to
   ensure that a single entity did not miss a packet.  By listening to
   all the \gloss{CHEEZBURGER}s on the network, a particular entity
   should be able to determine if they missed a packet from someone
   else.

   On top of this, dealing with RSA encryption keys is unwieldy at best.
   The entire purpose of having an ID hash represent each individual is
   precisely because simply using the public key as an identifier has
   far too great overhead.  So, retaining an accurate and up-to-date
   mapping of ID hash $\rightarrow$ public key is necessary.

   Additionally, Tube Truck has to be robust in the presence of heavy
   network congestion.  Without the traditional one-to-one form of
   Acknowledgements between a pair of clients, this becomes very
   difficult to guarantee to the application layer.  Determining a
   congested network from a dropped client hasn't been fully thought
   through.

   \section{Conclusion}
   \gloss{TTP} is a very sophisticated set of tools for the
   multicast application developer.  \gloss[short]{TTP} allows the
   application developer to focus on their task, rather than on trying
   to understand the complex web of multicast, RPC, and encryption that
   allows it to operate.

   That said, \gloss[short]{TTP} is an extremely young protocol with
   much room to grow.  It already has several known flaws, and has
   proven to be indeterminately difficult to implement because of the
   amount of state involved.

   Nevertheless, the application-facing ease of use makes the API an
   interesting place for a low-budget software developer to build on top
   of in order to leverage some of the more interesting features of IPv6
   Multicast.  The ability to develop every client in exactly the same
   way, with no need to distinguish roles (client/server) is highly
   appealing.

   \clearpage
   \appendix

   \section{Packet Types}
   \label{sec:pckts}
   \begin{figure}[htp]
      \begin{center}
         \includegraphics[width=\textwidth]{pictures/Pkt1}
      \end{center}
      \caption{Lobby Packets}
      \label{fig:p1}
   \end{figure}


   \begin{figure}[htp]
      \begin{center}
         \includegraphics[width=\textwidth]{pictures/Pkt2}
      \end{center}
      \caption{In-Group Packets}
      \label{fig:p2}
   \end{figure}

   \clearpage

   \setlength{\bibsep}{.1em}
   \bibliography{paper,rfc}
   \addcontentsline{toc}{section}{References}
   \clearpage
   \printgloss{gloss}

\end{document}
